A METHOD OF SELECTING TARGETS AND GENERATING 
FEEDBACK IN OBJECT TRACKING SYSTEMS 



BACKGROUND OF THE INVENTION 

The present invention relates to an improved 
video capture and display system. 

Object tracking systems are used for auto 
focusing, auto zoom, and other functions of video 
cameras. Likewise, object tracking systems are used for 
auto zoom, brightness adjustment, contrast adjustment, 
and highlighting functions of video display systems. 
Imaging devices in video cameras capture images by 
converting light into electrical signals through a 
photoelectric process. On the other hand, video display 
devices utilize electrical signals to control a light 
source producing an image. Object tracking systems use a 
number of known methods to process the electrical signals 
corresponding to an image and to extract a group of 
signals representing a designated target of the object 
tracking system. The group of signals representing the 
target object is relocated and "followed" in the 
succession of- images or frames making up a video 
sequence . 

To initiate the tracking process, a target must 
be initially identified for the object tracking system. 
Object tracking systems may automatically identify a 
target object based on an assumed usage model for the 
video capture or display device. For example, the object 
of interest will most likely to be located near the 
center of the image and may be the larger object in the 
vicinity. However, automatic target designation is 
complex and is problematic if the object of interest is 
not centrally located in the image, is one of a plurality 
of similar objects in the vicinity, or if there is little 
contrast between the object of interest and the 
surroundings . 
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User designation of the target object avoids 
some of the limitations and complexities of automatic 
target designation. Nishimura et al . , U.S. Patent 
No. 5,631,697, for VIDEO CAMERA CAPABLE OF AUTOMATIC 
5 TARGET TRACKING, describe a system in which the user 

designates the target by aligning an electronic marker 
generated in the viewfinder with the image of the 
intended target object and then depressing a switch to 
activate the target tracking processor. While less 
10 complex and more flexible than fully automatic target 
designation, the system provides only for user 
t3 designation of the target object's location, 

fj* To separate the group of signals representing 

CJ the target object from the larger group of signals 

U 15 representing the image, target tracking systems utilize 

45 information about the size of the target object, as well 

w as its location. Automatic or manual methods may be used 

fi to designate the target's size once its location has been 

|^ manually designated. Automatic determination of the 

ffl 2 0 target object's size is complex and often unreliable, 

0 especially in crowded or low contrast images. To reduce 

w the complexity of the tracking system and increase the 

reliability of target identification, the system may 
permit the user to designate the target's size. For 
25 example, the target object's size may be designated by 
drawing an "electronic" box bounding the target object. 
However, drawing a box around a moving target while 
operating a video camera is not easily accomplished and 
even locating a target in a viewfinder when there is 
30 little contrast between the target object and the 

surroundings or when the target is located at a distance 
may be difficult for a user. 

What is desired, therefore, is a method of 
designating a target object for an object tracking system 
35 that allows the user to easily locate the target object 
and reliably designate its location and size for the 
tracking system. Further, a method of notifying the user 



of the continuing performance of the tracking system is 
desired . 

SUMMARY OF THE INVENTION 

The present invention overcomes the 
aforementioned drawbacks of the prior art by providing an 
object tracking system that includes initiating the 
object tracking system, magnifying the image, selecting 
an object of interest in an image, and designating the 
object as the target all while the object tracking system 
is activated. In this manner it is easier for the user 
of the object tracking system to select the target, 
especially if the target tends to be moving fast. 

In another aspect of the object tracking system 
an image is magnified and an object is designated as the 
target for tracking by the tracking system. In response 
to designating the object the scale of the image is 
automatically changed. In one embodiment, this permits 
the user to zoom in on the image, select the target more 
easily, and then automatically zoom out, which permits 
easier selection of the target. 

In yet another aspect of the prevent invention, 
the object tracking system monitors a level of confidence 
that the tracking system is properly tracking the target . 
In response to a change in the level of confidence the 
magnification of the image visible to the operator is 
altered . 

The foregoing and other objectives, features 
and advantages of the invention will be more readily 
understood upon consideration of the following detailed 
description of the invention, taken in conjunction with 
the accompanying drawings . 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

FIG. 1 is a block diagram of a video camera 
employing the method of the present invention. 
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FIG. 2 is a flow chart of object tracking 
functionality. 

FIG. 3 is an exemplary viewf inder image of the 
video camera of FIG. 1. 
5 FIG. 4 is a magnified view of the exemplary 

viewf inder image of FIG. 2. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring to FIG. 1, a video camera according 
10 to the present invention includes video capture and 
display systems. The video camera includes a 
^ lens unit 10 , an imaging device 12 which may be a charge 

Ul coupled device (CCD) or metal oxide semiconductor (MOS) 

^ array, an A/D (analog to digital) converter 14, a signal 

=5 15 processor 16, a driver 18 to drive the imaging device 12, 

^ and a D/A (digital to analog) convertor 20. The imaging 

J"* device 12 includes a plurality of photoelectric elements 

|3 arranged in a two-dimensional array on one surface. The 

light of an optical image is focused on the photoelectric 
CI 20 elements of the imaging device 12 by the lens unit 10. 

Each photoelectric element converts the light from the 
image that is striking the element into an analog 
electrical signal. Periodically, the driver 18 connects 
elements of the imaging device 12 to the A/D converter 14 
25 causing the electrical signals produced by the elements 
of the imaging device 12 to be converted to digital 
signals which are read by the signal processor 16. The 
signal processor 16 converts the digital signals to video 
signals representing the luminance and chrominance of the 
30 light striking each element of the imaging device 12. 
The composite video signals are output to the D/A 
converter 20 for conversion to the analog signals of the 
camera's video output 22. The video output 2 2 may be 
transmitted or recorded and is displayed on a 
35 viewfinder 24 so that the user can visualize the images 
being transmitted or recorded by the camera. 



The video camera also includes a target 
tracking processor 26 that processes the output of the 
signal processor 16 to extract signals representing the 
target object from the signals representing the image in 
5 the succession of images or frames making up the video 

sequence. The video camera includes a zoom processor 28, 
a cursor generator 30, and a controller 32. The camera 
may also include a zooming motor 34 and an auto focus 
motor 3 6 driving the zooming and focusing mechanisms of 
10 the lens unit 10. As the lens 10 zooms in or out, the 
optical image is magnified on the imaging device 12 
producing a magnified image at the video output 2 2 and in 
the display of the viewfinder 24. It is to be understood 
that magnification may include the result of zooming in 
15 or zooming out. On the other hand, zooming may be 

accomplished by processing signals from a region of the 
imaging device 12. The zoom processor 28 may extract 
signals from a portion of the imaging device 12 and 
calculate zoom information from those signals to 
20 electronically magnify a portion of the image that is 
displayed in the viewfinder 24 . 
sub & ^ \^ Traditional video tracking systems include the/ 

user zooming in or zooming out until the object is 
clearly visible within the display and then the usj 
25 initiates the tracking functionality. Thereafter, the 
user then selects the target to be tracked by the video 
tracking system. The present invent or^tfame to the 
realization that while a wor kabl e>^chni que for tracking 
targets, it is clumsy because^xBy the time the user 
30 initiates the tracking fufr^t ionality and then selects the 
target, the target may'nave already moved out of the 
field of view. Th£s is especially true when the object 
takes up a s:Ujirff icant portion of the display, such as 
when unde^ext r erne zoom. To overcome these limitations 
3 5 the pj^sent inventors have developed a new interface for 
a tracking system. Referring to FIGS. 2 and 3, first the 
racking functionality is initialized by any appropriate 



technique. Next, the target is located, sized, and 
positioned on an electronic viewfinder 24 for accurately 
selecting and subsequent tracking. A cursor 42 is / 
generated by the cursor generator 3 0 and is superimposed 
5 on the image in the electronic viewfinder 24. Th^ 

cursor 4 0 may be in a fixed location in the vi^f inder 24 
(e.g., centered) and positioned over the imager of the 

target object by movement of the camera whic^h effectively 
moves the viewfinder image behind the cursor 42. On the 

10 other hand, the cursor may be movable and/ electronically 
positioned in the viewfinder image by operation of a 
joystick, touch pad or other pointing y&evice 38 (refer to 
FIG. 1). A target designator switch /3 9 facilitates 
signaling the controller 32 that a t/arget is being 

15 designated for tracking. The tarciet designator switch 3 9 
and the pointing device 3 8 may be/ combined into a single 
control element permitting the i/ser to conveniently 
position a movable cursor 42 aAd designate the target. A 
movable cursor 42 permits the' user to designate target 

2 0 objects that are not centered in the image. To initiate 

tracking, the user aligns /he cursor 42 to superimpose 
the cursor 42 on the target object and actuates the 
target designator switcM 39. Preferably, the target 
designator switch 39 aftd the pointing device 38 are 
25 combined by touch sensitive viewfinder 24 . In this 

manner, the user ma./ simply target and select the target 
to be tracked by a/ single action. The physical activity 
used on the videc/ camera for targeting and selecting 
(e.g., buttons /nd controls) are preferably different 

3 0 than that usedr for zooming, which permits the targeting 

and selecting functions to be placed into a "ready" state. 
In the case/of the touch screen viewfinder 24, while 
touching tfoe viewfinder 24 may be used for many different 
function/, after initialing the tracking functionality 
35 the touch screen viewfinder ! s next function is 

automatically set to targeting and selecting of the 
target. Accordingly, after zooming a single action of 
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touching the viewfinder 24 screen^i-li-perfofm targeting 
and selecting in a fa^Jt--«iT9ef f icient manner before the 
target, espe«-rSTly if moving fast, is no longer displayed 
on viewfinder 24 . 

In one embodiment of the present inventior 
actuation of the target designator switch 3 9 causes/the 
controller 36 to signal the zoom processor 28 to >Degin 
zooming. The zoom processor 2 8 performs the ^om 
calculation or actuates the zoom motor 34 casing the 
zoom lens unit 34 to zoom in, magnif ying^che image in the 
viewfinder 24. Referring to FIG. 4 , fcii e image in the 
viewfinder 24 is magnified until t&e target object 44 is 
easy to view and designate. The user then designates the 
target object by actuating ttfe target designator switch 
3 9 for a second time. Th^s may likewise be performed by 
a touch sensitive viewfinder 24 where after initiating 
tracking functionality the user touches the screen to 
start zooming and/couches the screen a second time to 
designate the target. The first zooming may be 
automatic, ify^Lesired. Designation of the target causes 
the target tracking processor 2 6 to extract the signals 
representing a target designated at the location of the 
cursor (^r touch on the screen) from the signals of the 
image ijci a scan of the imaging device 12 . A target may 
be idgoitified in any manner. The target tracking 
processor 2 6 then "follows" this group signals 
representing the target object in subsequent scans of the 
imaging device 12 to track the target as the video 
sequence progresses . 

- x In another embodiment of the present invent ia 



actuation of the target designator switch 3 9 causes" the 
controller 36 to signal the target tracking processor to 
initiate tracking and the zoom^p-rdcessor 28 to begin 
zooming in. When the imatje of the tentative target is 
easy to view in the viewfinder 24 the user designates the 
target ob j eofe^a second time by actuating the target 
designa^efr switch 39. The target object is tentatively 
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identified for the target tracking processor 
first actuation of the target^esigii^orswitch 39 but 
the information abQy£r--irFie size and location of the target 
is refined 3x*dT verified in the magnified image where the 
taro^tr^obj ect is more visible. 

The zoom processor 2 8 may then zoom out 
electronically or by actuating the zoom motor 34 to 
return the zoom to the original setting or some 
intermediate setting established by the user. The 
10 present inventors realized that zooming in on the target 
object would make it easier to locate the target object 
if it is at a distance from the camera or in low contrast 
or crowded surroundings. Further, automatically zooming 
the image provides a convenient method of establishing 
15 the target. This may likewise be performed with a touch 
sensitive display . 

The algorithm applied by the target tracking 
processor 2 6 may include a measurement of the level of 
confidence that the target is being correctly tracked. 

2 0 If the level of confidence should decrease, the tracking 

processor 2 6 may cause the zoom processor 2 8 to begin 
zooming in. The changing magnification of the image in 
the viewfinder 24 notifies the user that the confidence 
level of the' algorithm is low. Increasing the 

25 magnification of the image may cause the algorithm's 

confidence level to increase. On the other hand, if the 
algorithm cannot automatically reacquire the target 
object, increasing the magnification of the image will 
facilitate relocation of the target object by the user 

30 who may re-designate the target for the tracking system 
by actuating the target designator switch 39. 
5~V ^ §^ \ while tracking a target periodically the targets 

may be obscured or otherwise not visible oil 
viewfinder 24 for a period of -- J^im^T - This may occur, for 

3 5 example, when another>ei5]ect comes in between the target 
and the camer^*t5r a period of time. When the target is 
not det^tTed by the target tracking processor, the system 



may automatically switch into an alternative global,^^ 
tracking motion mode to predict where the oj^g-etft should 
be. In this manner, when the target^irs^fio longer 
obscured or otherwise not v^iirle on the viewf inder 24 
the target trackinojpafx5cessor will have an increased 
change of idepfe-drfying the target to continue tracking. 
In this manner, the system is less likely to loose track 
of tjare target. 

When the confidence of the system is 
sufficiently low (or losing the target totally) the 
present inventors came to the realization that most 
likely the target is no longer visible within the 
viewf inder 24 . Accordingly, in such an event the system 
may automatically zoom out (or in) to present the viewer 
with an image having a greater (or lesser) field of view. 
In most instances, the target will be visible to the user 
within the new field of view image. The user then may 
select the target within this field of view in the manner 
described above, as desired. Alternatively, after the 
zooming in response to the confidence level the system 
may exit the tracking functionality. 

Yet in another aspect of the present invention, 
the present inventors came to the realization that 
control over the amount of magnification provided by the 
system for the selection of the target should operate in 
an intuitive and efficient manner. The intuitive system 
developed by the present inventors includes the user 
selecting a target object for tracking, in any manner, 
such as those previously described. Preferably, after 
selecting the target object the tracking system tracks 
the target object. In the event that the user triggers 
another target object selection within a predetermined 
time period, then the system will cease object tracking, 
if object tracking was already initiated, and change the 
magnification of the image on the display. The same or a 
different object may be selected for the another target 
object selection. For example, the magnification for the 
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target objection selection may be performed by, for 
example, clicking a mouse twice (or multiple times) or 
touching a display twice (or multiple times) . After 
changing the magnification of the image, the system then 
5 permits the user to select a target object for tracking. 
Again, if the user triggers another target object 
selection within a predetermined time period, then the 
system will cease object tracking, if object tracking was 
already initiated, and change the magnification of the 

10 image on the display (such as increase magnification) . 

This process is repeated until the user does not trigger 
another target object selection within a predetermined 
time period. This permits magnification of the image 
until the target can be easily selected for tracking. 

15 The target selected for tracking is preferably the last 
selected object. After tracking is initiated, and 
preferably the predetermined time period has elapsed, the 
system may return to a previous magnification, if 
desired . 

20 While the tracking system has been described in 

relation to a video camera, the teachings are equally 
applicable to selection and tracking when viewing video, 
from any source, such as for example, a video cassette 
recorder, digital tape, analog tape, compact disc, and 

25 digital video disc. 

All the references cited herein are 
incorporated by reference. 

The terms and expressions that have been 
employed in the foregoing specification are used as terms 

30 of description and not of limitation, and there is no 

intention, in the use of such terms and expressions, of 
excluding equivalents of the features shown and described 
or portions thereof, it being recognized that the scope 
of the invention is defined and limited only by the 

35 claims that follow. 



